Video cameras, in particular video cameras which are used for monitoring purposes, should provide a large volume of image data with the power consumption being as low as possible. However, the data processing tasks are significant. They include, i.e., the preparation of the raw data from an image sensor, for example, the generation of a color image on the basis of the color image recorded by using a Bayer filter, denoising, the performance of gamma corrections, the compression of individual images, e.g., by means of JPEG algorithms, the M×PEG method and the like, the compression of image data video streams in view of image parts that are not of interest, image parts with at the most marginal changes in the long run and so on, the perspective rectification, the rectification of fish eye lens images in connection with the use of wide-angle lenses for monitoring purposes, the selection of image details and so on.
In particular in connection with high-resolution sensors, the data volumes to be processed are often quite significant. At the same time, however, the same image processing steps have to be performed again and again as long as there are no severe changes. For example, the calculations for perspective distortions remain unchanged as long as the viewing direction and the selected image detail do not change. Furthermore, denoising algorithms, gamma corrections and so on are almost the same for a plurality of images as long as the light situation does not change or changes at the most little. Thus, it makes sense per se to adapt the data processing in an optimum manner in accordance with the recurring processing steps that are the same during a relatively long time period.
For carrying out always the same processing steps with data of a data stream, FPGAs are particularly suitable. In modern FPGAs, a plurality of coarse-granular arithmetic and logic units (ALUs) are present, by means of which a plurality of processing steps can be carried out in parallel and subsequently. For this purpose, the function of each ALU is determined once, i.e., “configured”, and the intermediate results obtained in a step-wise manner are passed on from ALU cell to ALU cell in the FPGA.
It is pointed out that local feedback loops, for example, for realizing MAC functionalities etc., can be present and that local registers or memory spaces can be used. As an example only, reference is made to modern FPGA architectures such as the Virtex series of the company Xilinx which have been available at the filing date. The use of FPGAs for the use of data streams is advantageous in particular because it is not necessary to load and decode a new command for each processing step as is the case in sequential processors.
However, it should be taken into consideration that when the desired data processing of the data stream changes, a new configuration of the FPGA must be provided. This is called reconfiguration and there are techniques for reconfiguring the FPGA such that the reconfiguration needs only little time and can possibly and preferably take place partially, i.e., by maintaining the configuration on the other FPGA parts which should presently maintain their functionality, i.e., should not be reconfigured.
From the essay “Internal dynamic partial reconfiguration for real time signal processing on FPGA” by S. U. Bhandair, Indian Journal of Science and Technology, vol. 3, no. 4, April 2010, ISSN 0974-6846 it is known to process audio-video data by means of an FPGA that can be reconfigured at runtime. It is stated that a present sequential processor can configure the audio and video filters in accordance with the selection of a user by loading respective partial configuration bit streams through a specific port into the FPGA. Achievable configuration times are determined. It is pointed out that irrespective of the audio and video filters used for description purposes, also other applications should at the same time profit from the described kind of reconfiguration via the specific port. According to other research, see B. Draper, R. Beveridge, W. Böhm, C. Ross and M. Chawathe in “Accelerated Image Processing on FPGA's”, IEEE Transactions on Image Processing, 12(12):1543-1551, 2003, a considerable acceleration as compared to sequential processors can be achieved by using FPGAs. Programming difficulties are said to be reduced by a programming language in which i.e. pointers and recursions are forbidden and variables are assigned only once. It is discussed that stand-alone-applications on FPGAs are desirable, e.g., in a camera for security applications, for monitoring images in view of irregularities and for informing users via the Internet. Movement detection and face recognition are mentioned.
From the document A. Herkersdorf and W. Stechele: “AutoVision—Reconfigurable Hardware for video-based Driver Assistance Systems”—TU München, September 2009, available under URL: https//www.12.informatik.uni-erlangen.de/spprr/colloquium09/autovision.pdf, it is known to prepare video data for driver assistance systems by means of FPGAs, wherein for the FPGAs a dynamic parallel reconfiguration is discussed as inter or intra video frame reconfiguration. It is mentioned that specific conditions such as daytime and nighttime require different evaluations and the influence of the reconfiguration time is discussed.
US 2008/0098211 A1 discloses the dynamic reconfiguration of data-processing circuits, wherein for achieving a partial reconfiguration, a memory is provided which comprises information for an alternative configuration which should be linked with original configuration information.
From the essay “Real-time correction of distortion image based on FPGA” by H. Zheng and J. Lee in International Conference on Intelligent Computing and Integrated System (ICISS); IEEE Conference Publication, 2010, pages 167-170, it is known to prepare video image data in a block-wise manner by means of an FPGA for achieving a distortion of an infrared camera image. The information required for the rectification is said to be stored in a lookup table in a ROM.
The essay “3-D Perspective Video Scaling Effects on FPGA” by Eivind Karlsen, Norwegian University of Science and Technology, 2013, discloses the use of a module for converting video signals between two different resolutions (video scaler) for carrying out a perspective transformation of a video stream. It is discussed that depending on the implementation of the algorithms, different image qualities can be achieved. Moreover, two different calculation methods are mentioned, namely “forward mapping”, wherein the position of each input pixel is calculated in the output image and a frame buffer is necessary for the output for storing the output image during its generation and, on the other hand, the “reverse mapping”, wherein each output pixel is calculated sequentially.
From the essay by Pierre Greisen et al. “An FPGA based processing pipeline for high-definition Video”, Eurasip Journal on Image and Video Processing 2011, 2011:18, the video-image processing for endoscopic video recordings is known. FPGAs are said to be used for carrying out, i.e., image rectification and an estimation of inequalities in real-time and with full HD-resolution. These inequalities are said to be based directly on differences in the camera views resulting from the actual geometry of a scene. It is mentioned that specific corrections such as color corrections and rectification corrections are particularly important in stereo recordings for achieving an improved image reproduction. The over-linear increase in the processing complexity with increasing video resolution is mentioned. The FPGA is said to be integrated in a heterogeneous PC platform which also comprises a CPU and a GPU. It is suggested that the CPU and the GPU resources are used for operations such as control algorithms for the system settings or for storage-intensive algorithms such as segmenting, which are ill-suited for FPGA processing. An architecture is suggested in which first sensor noise (pattern noise) is corrected and different pixel sensitivities are compensated for. Then the Bayer demosaicing is said to take place, a linear color correction is carried out in accordance with a selected color space. It is stated that these processing steps can be carried out in a pipelined manner on small data windows. It is stated that the video streams coming in from two cameras for 3D purposes can be temporarily stored in small FIFOs for the purpose of synchronization, which is said to be advantageous in case of small deviations of the camera clocks. It is also mentioned that it can be advantageous to temporarily store data in a sufficiently large cache.
From the essay “Low-Cost real-time depth perception with FPGAs” by B. Borosky, Microtechnology Europe, April 2008, pages 28 et seqq., it is known to correct fish eye images by means of FPGAs by linking the image pixels by using a lookup table (LUT).